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ABSTRACT 

This paper compares two methodologies for selecting 
peer institutions used by the National Center for Higher Education 
Management Systems (NCHEMS) and by the State of Kansas. The 
Un i vers ity^of Kansas was used as a test institution to compare the 
NCHEMS and Kansas methodologies. Attention is directed to differences 
in results, the impact of data availability and data quality on 
results, and. whether one methodology provides a better set of peer 
institutions than the other according to descibed criteria. The 
procedure used most often at NCHEMS for selecting a group of 
comparable institutions is based on criteria established by the home 
institution. After selecting institutional characteristics for 
judging similarity, institutions are rank-ordered by their 
similarity. The Kansas methodology was developed to quantitatively 
assess earlier peer selections that were based on informed judgment. 
Categories used to determine institutional similarity are 
enrollments, funding and expenditure patterns, and degree level. 
Appendices include: a list of characteristics used at NCHEMS to 
select peer groups for four-year institutions; a list of factors and 
weights used to determine peer groups fpr the University of Kansas 
using the Kansas methodology, and a list of strengths and weaknesses 
of the two methodologies. (SW) 
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A COMPARISON OF PROCEDURES 
FOR ESTABLISHING PEER GROUPS 

Abstract 

The absence of standards to evaluate financial problems and increased emphasis on 
accountability has caused increasing pressure to use comparative data to establish 
norms for evaluating programs and budgets. Typically administrators want a set of 
"peer" institutions for these purposes. A number of methodologies have been developed 
and this paper will contrast two of them. 

Introduction 

The pressures for institutions to use external sources as a means to rationalize 

► 

their activities continue as a result of financial stress and an emphasis on 
accountability. The lack of any absolute standard* or frame of reference for 
evaluating institutional performance is also a contributing factor. It is not known 
how broad the curriculum should be at a certain type of institution, or how much the 
cost should be to produce a given number of credit hours, or what percentage of an 
institution's budget should be spent on library or other services. In the absence of 
standards, administrators turn to the behavior of other institutions, either 
individually or as a group, to establish norms for guidance. 

Typically administrators want a set of "peer" Institutions— institutions quite 
similar to their own— for planning, resource allocation, and performance measurement 
purposes (Terenzlni, et al., 1980} • There are many methodologies available for 
determining an institution's peer group including those developed by the American 
Association of University Professors (AAUP}, the Carnegie Commission for Higher 
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Education, the National Center for Higher Education Management Systems (NCHEMS), and 
by individual states such as Kansas and Washington. 

Each of the methodologies for identifying peer institutions uses different 
criteria but usually includes some subset of the following variables: enrollment, 
number of degrees earned, programs offered, professional staffing, average salaries, 
and research expenditures, among others. The extent of the differences of these two 
methodologies in producing a set of peer institutions is not known. . 

As external agencies (coordinating boards, state budget offices, legislatures) 
increasingly rely on peer data for evaluating programs and budgets, institutions are 
growing more concerned about the selection of peers. This paper will compare and 
contrast the methodologies developed and used at NCHEMS and by the State of Kansas for 
selecting peer institutions. ' Some of the questions to be addressed will be: 1) are 
there differences in the results and to what extent, ^j^what is the impact of dtta 
availability and data quality on the results, and 3) does one methodology provide a 
better set of peer institutions according to the desired criteria. 

Procedures 

This section describes the methodologies developed and used by NCHEMS and by the 
Kansas Regents for selecting peer institutions. 

NCHEMS 

The procedure used most often at NCHEMS for selecting a group of comparable 
institutions is based on criteria established by the "Home* institution— the 
institution searching for a peer group. The first sU .. this procedure is to 
determine which institutional characteristics should be used to establish similarity. 
In most instances institutional mission is used as the basis of selecting the 
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characteristics. Table I displays a list of characteristics typically used at NCKEMS 
for 4-year colleges or universities for selecting peer institutions. The nominal 
variables (set 1) are used as selection criterion whose purpose is to reduce the 
relevant universe of institutions. Institutions are asked to indicate the importance 
of each criteria, and for each response to an item checked "very important/ any 
non-matching institutions are eliminated from further consideration. 

The variables in set 2 of table 1 are interval variables used to move 
institutions up or down on a list of possible comparison schools. Based on a set of 
ranges established by the home institution for each of the variables, a candidate 
institution will either land in or out of the ranges established. The more frequently 
an institution is outside the ranges that are established, the further down the list 
it is placed. In addition, a weighted score 1s calculated, using the importance 
scale. A miss counts one point if the variable is "very important," one-half point 
for "Important," ami no points are added for a miss on an "unimportant" variable. The 
weighted sum is then used to rank-order the candidate institutions. Thus, an 
institution's place on the list will be a function of both how well it fits the 
criteria and the weight assigned to those criteria. 

On the basis of the criteria established, a list of institutions rank-ordered by 
their "closeness" is established (refer to table 2 for an example). NCHBS recommends 
that a subset of the list— approximately 15-20 institutions— should be selected as the 
comparison group for whatever further analysis is Intended. Intervention on the part 
of the home institution analyst is critical because the rank-ordering program ignores 
the extf >t to which a candidate institution misses a range. A single very large miss 
might be sufficient reason to disqualify a candidate institution from further 
consideration, even if it did well on the other comparison dimensions. Furthermore, 
intervention is necessary because the NCKEMS approach is no| designed to be a kind of 
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.0 


.0 


8060 
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13.6 


0.0 


0.0 


11.2 
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.013 
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4 


.0 


.0 


8457 


773 


22.3 
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0.0 


0.0 


72.8 
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22.1 
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.0 


.0 


6162 


S9.9 


10.1 


0.0 


0.0 


0.0 


73.5 


.019 


21.2 
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.0 


.0 


8210 


87.1 
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0.0 


0.0 


2.3 


66.1 


.021 


.15.8 


7 


.5 


1.0 


COW 


74.2 


15.7 


0.0 


0.0 


10.1 


69.1 


,.010 


32.0 
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.5 


1.0 


9789 


S4.S 
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0.0 


0.0 


55.9 


.036 


9.1 
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8104 


82.7 
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0.0 


4.8 
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4430 


83.8 


12.5 


0.0 


0.0 


3.8 


78.8 


/ .002 


22.8 


IS 


1.0 


1.0 


5468 


81.9 


18.0 


0.0 


0.0 
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80.2 


.005 
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1.0 


4754 
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12.0 


0.0 


0.0 


4.8 
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.041 


16.7 
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1.0 


4946 
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0.0 


0.0 
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.008 


12.3 


18 


1.0 
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2.0 


69S4 


80.4 
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0.0 


0.0 


0.0 
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.018 
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90.9 
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0.0 


0.0 
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.041 
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0.0 
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.002 
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"turn-key" system for generating peer groups. Rather, it provides a quick way to find 
the relevant set of institutions in the H£SIS universe from which a peer group might 
reasonably be selected. 

Kansas k< 

The Kansas methodology was developed in 1978 to quantitatively assess earlier 
peer selections which were based upon Mnfonaed judgments." 

To initially identify candidate ^institutions, the Kansas methodology allows 
selection from specified states, control of institution (public, private, 4-year, 
2-year, etc.), number of doctoral programs offered at the two-digit HEGIS (Higher 
Education General Information Survey) taxonomy level of the institution, or any other 
characteristic recorded in the KEGZS institutional characteristics survey. 

After candidate institutions are identified, three types of institutional 
characteristics are considered to measure similarity: (1) enrollment, (2) funding and 
expenditure patterns, and (3) degree programs. These characteristics and, examples 
of relative weights used to determine institutional similarity are further detailed in 
table 3. Means and standard deviations are calculated for each variable. Deviation 
scores (z) are calculated using the formula z^Xj-x/C. The transformation of these 
raw data to z-scores allows further comparisons and manipulations. 

A comparison score (c) between z-scores for the home institution and the 
candidate institutions is calculated by taking the absolute value of their 
differences. To compare degrees conferred , a matrix of degrees by two-digit HEGIS 
areas at four degree levels (bachelor, master, doctoral , and first professional) is 
generated. In analyzing degrees, a mean and standard deviation is found for each cell 
of the matrix. Comparison scores are then calculated and aggregated by degree level 
and divided fay the number of two-digit HEGIS areas where degrees are conferred by both 
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Table 3 



Factors and Relative Weights Used to Determine Institutional Similarity 

in Kansas Methodology 



Characteristic Factor 



Relative Weight 
Example I Example 2 Example 3 



Enrollment 



Financial 



Bachelor's 
degree 



Master's 
degree 

Doctoral 
degree 

First 

professional 
degree 



Full-time equivalent enrollment 
Headcount enrollment 
Graduate enrollment as a percentage 
of total enrollment 

Instruction expenditures as a 

percentage of total E&6 

expenditures 
Research expenditure as a percentage 

of total £18 expenditures 
Public service expenditure as a 

percentage of total ESQ 

expenditures 
Other expenditures as a percentage 

of total £&a expenditures 
Restricted use funding as a percentage 

of total funding ; 

Percentage of all bachelor's 
degrees conferred in each 
academic field (two-digit 
KEGIS category) J 

Percentage of all master's degrees 
conferred in each academic field 

Percentage of all doctoral degrees 
conferred in each academic field 

Percentage of all first professional 
degrees conferred in each academic 
field 







1055 


5 


10 


5 


15 


15 


15 


2 


2 


2 


2 


2 


2 


2 
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2 


2 


2 
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2 


2 


30 


30 


30 


5 


20 


25 


20 


10 


5 



( 
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the home institution and the candidate institution. This procedure results in four 
values for degrees. 

The comparison scores (c) are standardized using the formula: x^ « 50 ♦ (10 * 
C j). Since z-scores commonly fall in the range of -3 to 3, this transformation causes 
the Comparison scores to become nonnegative win. a broader range. Weights (totaling 
100) are then applied to the standardized comparison scores and summed to create 
similarity scores. Since the comparison score for a home institution is zero, this 
process results in a similarity score of 5,000 for the home institution. According to 
their similari y scores, institutions are rank-ordered for the home institution. 
Table 4 presents a sample listing of the output using this methodology. 

Comparison of Results 

The University of Kansas (KU) was used as a test institution to compare the 
NCHEMS and Kansas methodologies to select peer institutions. Refer to appendix A for 
the criteria used for eacn of the methodologies. 

i i 

A comparison of the results of the two methodologies (appendix B) reveals that 
among the top ten ranked institutions, seven of the institutions are the same although 
the rank order differs. Further analysis reveals that two of the three institutions 
that appear for Kansas but not for KCKEMS (#4-ranked University of Houston and 
#5-ranked Wayne State University) rank among the first 25 institutions on the NCHEMS 
listing. Similarly, two of the top ten Institutions that appear on the NCHEMS list 
but not on the Kansas listing, rank among the top 25 institutions for Kansas. The 
institution that KU considers very. important to have as a peer institution. University 
of Colorado—Boulder (CU), ranked 42nd on the NCHEMS listing. 
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Table 4 



Peer Analysis - Summary Rank Qr4er 
Kansas Methodology 



Enrollment Weights 

FTE 10.0 
Keadcou.it 5.0 
% Graduate 15.0 



Financial Weights 

Instruction 2.0 

Research 2.0 

Service 2.0 

Other 2.0 

Restricted 2.0 



Degree Heights 

Bachelors 30.0 

Masters 5.0 

Doctoral s 20.0 

First Prof 5.0 



institution Name 



Rank 

1 University of Kansas (Main Campus) 

2 University of South Carolina at Columbia 

3 University of Colorado at Boulder 

4 University of .Houston (Central Campus) 

5 Wayne State University 

6 University of Oregon (Main Campus) 

7 University of New Mexico (Main Campus) 

8 University of Cincinnati {Main Campus) 

9 University of Oklahoma (Norman Campus) 

10 University of Iowa 

11 Indiana University at Blooming ton 

12 Arizona State University 



Score 





000 


5602. 


,477 


5618, 


.25S 


5637. 


.188 


5660. 


.352 


5661. 


.609 


5681. 


.733 


5697, 


.770 


5768. 


.552 


5779, 


.605 


5813, 


.145 


5814, 


.754 
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In order to try to get CU among the top ten Institutions in the NCHEMS listing, 
the criteria were changed slightly. Append*.- A- 3 gives the criteria for the second 
iteration. A new listing (appendix C) was produced with the results that eight of the 
ten institutions were the same for both iterations although the two sets of criteria 
differed slightly. In the second listing CU ranked number two. 

The major difference between these two methodologies is that in this example, the 
NCHEMS criteria used the proportion of degrees awarded by level and the proportion of 
degrees awarded by discipline as separate variables, whereas Kansas used the 
proportion of degrees awarded by level by discipline. For example, in the NCHEMS < 
analysis, several Institutions that ranked in the top 20 institutions statistically 
had ti e required proportion of graduate degrees, but the doctoral degrees were awarded 
only in two three fields. The Kansas methodology would have initially screened 
those institutions out for having "too few" doctoral programs . The same effect can be 
achieved with the NCHEMS approach. The criteria, number of doctoral programs , can be 
added to the list of criteria (table 1), and a range and an importance level 

« 

established. Or, a minimum number of doctoral programs can be set as a hardpoint 
requirement, removing from the list any Institutions who don't offer at least that 

Mi I It SUJUif* MfciiiftWWi » 

Conclusions 

'>> 

The strengths and weaknesses of the NCHEMS and Kansas methodologies are 
summarized in table 5. One methodology does not produce a "better" group than the 
other. In fact, the results show that the two methodologies produce very similar 
listings. 

One potential criticism of both methodologies i c that both rely ije-vily on HEGIS 
data which in itself has some shortcomings. However, the authors believe t|jat since 
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Tables 

A Sfawary of the Strengths and Weaknesses 
of the NCHEMS and Kansas Methodologies for Selecting Peer Institutions 



KCHEKS 

Strengths Ease of understanding 
by non-statisticians 

Ease of implementation 
Inexpensive to run 



Kansas 

Statistically sound methodology 
Inexpensive to run 



Mixed 
blessing 



Weaknesses 



Ease of manipulation can 
lead to game playing in 
certain political 
environments 



Arbitrariness of 
method logy 

V 

No factor for program 
quality 



Data detail permits extensive 
examination at a candidate institution, 
particularly degrees awarded by 
discipline and level 

Difficult to manipulate to 
achieve preconceived ideas 

Difficult to understand 
methodology by non- statisticians 

No factor for program quality 



u 
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HEGIS is the only comprehensive data set available, it can and should be used with the 
user being aware of its limitations. For further discussion on HEGIS data quality 
issues, refer to Firnberg and Christal (1984}. 

The purposes and desires of the home institution will determine which methodology 
might best serve its needs. For instance, if the program emphasis is important and 
essential, the Kansas methodology might be the preferred technique. Or, if the home 
institution wants to be able to easily explain the methodology to a governing board or 
a legislature, the NCHEMS methodology might be selected. Other, factors need to be 
considered in selecting peer institutions to increase the credibility. 

Institutions have long been accused of selecting peers to suit their 
needs— sometimes to appear underbudgeted and overworked when requesting funds, other 
times to reflect a group of institutions they aspire to be like. As peer groups are 
increasingly used by exteral agencies to evaluate programs and budgets, objective 
analyses that can withstand political scrutiny become more important. The 
methodologies discussed In this paper were developed in part to respond to these 
concerns. However, quantitative methods cannot account for all the factors that 
should be a part of a peer selection process— namely, qualitative aspects of an 
institution. Therefore, peers produced by quantitative methodologies must be further 
evaluated by subjective or "informed" judgments in selecting a final group of peers. 
Analyses of numbers are not a substitute for good judgment but rather should enhance 
and inform judgment. 
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Appendix A-l 



Factors and Weights Used to Determine Peer Group for 
University of Kansas Using the Kansas Methodology 



Characteristic Factor , 



Weight 



Enrollment 



Financial 




Bachelor's 
degree 



Master's 
degree 

Doctoral 
degree 

First 

professional 
degree 



Full-tin^ equivalent enrollment 
Headcount enrollment 
Graduate enrollment as a percentage 
of tofcM enrol Iment 

fnltructioA expenditures as a 
' -percentage of total ES£ 

expenditures 
Research expenditure as a percentage 
of total ESS expenditures 
Ubilc service expenditure as a 
percentage of total E&G 
expenditures 
Other expenditures as a percentage 

of total £&G expenditures 
Restricted use funding as a percent 
of total funding 

Percentage of all bachelor's 
degrees conferred in each 
academic field (two-digit 
HESXS category) 

Percentage of all master's degrees 
conferred in each academic field 

Percentage of all doctoral degrees 
conferred in each academic field 

Percentage of all first professional 
degrees conferred in each academic 
field 



10.0 
5.0 

15.0 



2.0 
2.0 

2.0 

2.0 
2.0 



30.0 
5.0 
20.0 

5.0 
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Appendix A-2 
^ITERIA FOR COMPARISON, 4-YEAR INSTITUTIONS: 
FOR University of Kansas (1st iteration) 



Set I. 



Your 

Items Institution 


] Check One 




Very \ 
Impt 


Not 
Impt 


MiHiHHi'imninnii—— 


Public 


X 




Undgrant no 


X 




Medical School no 




X 


Urban/rural Urban 




X 


Region 

(N.Atlantic/Gr.Lakes&PUins/ Great Lakes 
Southeast/West&Southwest } & Plains 




X 


\ • . 

Set II. 


/ 




\ 

Items \ 


Your 
institution 


Range 




Check one 




Very 
Impt 


Impt 


Not 
Impt 


Total FT£ Students 


20,3§4 : 


18-24 ,000 




x 




% Part-time Headcount Students 











X 


% AA Degrees 


0 


0-5 , 






X. : 


% BA Degrees 1 


65.6 


55-75 




1 X 




Jt'MA Dear«*es i 


24.5 


20-30 


X 






I~PHD Decrees 


5.5 


2-10 


X 






% 1st Professional Degrees 


4.3 


2-6 




X < 




% Graduate Degrees 




25-45 


| .. 






% MA & 1st Prof. Degrees 


28. § 


25-33 ' 




— a — I 




% Degrees in Prof. Fields 




eo-p? 








% Degrees In Non-Sci Prof Fields 


46.4 


40-50 


; X 






% Degrees in Engineering 


9.0 


5-15 




X 




% Degrees in Engineering Tech. 


0 


0 






! K 


% Degrees in Science 


9.8 


5-15 




X 




% Degrees in Science & Engin. " 


18.7 


10-30 




* i 




% Grad. Degrees in Sci & Engin 


5.3 


3-10 




X 




% Degrees in Business 


I 16.1 


10-20 




X 




% m Degrees in Business 




2-8 




X 




% Degrees in Education 


I 13.0 


10-20 




X 




% HA Degrees in Education 


S.3 


4-10 




* 




Research Expenditures/ 
Instruction Expenditures 


.- j>.37 




,-JS 






Research Expenditures ! 


$18. 2m 


l5-35m 








Research £xps Per FfE Student 


$?P7 




m 




— 






Appendix A- 3 
CRITERIA FOR COMPARISON, 4-YEAR INSTITUTIONS: 
FOR University of Kansas (2nd iteration) 



Set I. 



I Your 
Items Institution 


Check One 


.pry 
Impt 


Not 
Impt 




: X 




Landqrant no 


i x 




Medical School \ no 




I x 


Urban/rural \ Urban 




X 


Region \ 
(N.Atlantic/GrAakes&Plains/ Great Lakes 
Southeast/West&Southwest] & Plains , 




. ft ; 



I - ■ 

Set II. 



Items 


Your 
Institution 


Range 


Check one \l 


impt 


Impt 


TNot | 
Impt | 


Total FTE Students , 


- 20.364 


18-24,000 




X 




% Part-time Headcount Students 




20-SO 






x 1 


% AA Degrees 


0 


0-5 






X i 


% BA Degrees 


65. 




• 55-80 




X 




% MA Degrees 


24. 


.5 


15-25 


X 






X PHD Degrees 


5. 


,5 


3-7 


X 






* 1st Professional Degrees 


4. 


,3 


3-10 




X 




% Graduate Degrees 


1 34. 


.4 


20-4* 


X 






2 MA & 1st Prof. Degrees 


! 




18-35 




X 




% Oeqrees in Prof. Fields 


—■ SS. 


,1 




X 






% Deqrees in Non-Sd Prof Fields 


46, 


,4 


35-60 


X 






% Deqrees in Engineering 


— §: 


,0' i 


5-15 




X 




% Degrees in Engineering Tech. 


— ™ d 




0 






X 


% Deqrees in Science 


$ 


.d 


7-14 




X 




% Degrees in Science & Engin. 


~ IS, 


.7 


9-27 




X 




% Grad. Degrees in 561 & tngin 


5, 


.3 


3-7 




X 




% Degrees in Business 


rs". 


.1 


14-20 




X 




% m Degrees in Business 




.7 






X 




% Degrees in Education 


13; 


.0" 






X 




% MA Degrees in Education 


6' 


.3' 


J-ii 




X 




Research Expenditures/ 
Instruction Expenditures 


.37 


.25-. 55 


x • 






Research Expenditures 


Si&.SSi 


'"" il-35 " 




X 




Research Exos Per FTE Student 


"■ " $907 


1 508-1506 




X 





4 



Appendix 3 



NCHEMS 



Comparison of the Peer Institutions Using the NCHEMS* 

and Kansas Methodology 

Kansas 



1. University of Kansas 1. 

2. University of Oklahoma-Norman 2. 

3. University of Iowa 3. 

4. University of Utah 4, 

5. University of Cincinnati S. 

6. University of New Mexico 6. 

7. University of Oregon 7, 
3. Univ. of No. Carolina-Chapel Hill 8. 

9. Univ. of So. Carolina-Columbia 9. 

10. Virginia Commonwealth Univ. 10.. 



University of Kansas 
Univ. of South Carolina-Columbia 
University of Colorado-Boulder , 
University of Houston 
Wayne State University 
University of Oregon 
University of New Mexico 
University of Cincinnati 
University of Oklahoma-Norman 
University of Iowa 



* 1st iteration 



20 

17 



L 



Appendix C 

Comparison of the Peer Institutions Using the NCHEM5* 

and Kansas Methodology 



NCHEKS 



1. University of Kansas 

2. University of Colorado-Boulder 

3. University of Cincinnati 

4. University of Iowa 

5., Univ. of No. Carolina-Chapel Hill 

6. University of Oregon 

7. Florida State University 

8. Univ. of So. Carolina-Columbia 

9. Univ. of Utah 

10. University of New Mexico « 



Kansas 

1. University of Kansas 

2. Univ. of South Carolina-Columbia 

3. University of Colorado-Boulder \ 

4. University of Houston 

5. Wayne State University 

6. University of Oregon 

7. University of New Mexico 

8. University of Cincinnati 

9. University of Oklahoma- Norm an 

10. University of Iowa 



* 2nd iteration 



21 

18 

o 



